Model-driven evaluator bias detection

ABSTRACT

A method for detecting bias in an evaluation process is provided. The method includes operations of receiving evaluation data from a candidate evaluation system. The evaluation data is provided by a set of evaluators based on digital interview data collected from evaluation candidates. The operations of the method further include extracting indicators of characteristics of the evaluation candidates from the digital interview data, classifying the evaluation candidates based on the indicators extracted from the digital interview data, and determining whether the evaluation data indicates a bias of one or more evaluators with respect to a classification of the evaluation candidates.

RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/015,306 filed Jun. 20, 2014, and entitled “Model Driven EvaluatorBias Detection,” the entirety of which is incorporated herein byreference.

BACKGROUND

Finding and hiring employees is a task that impacts most modernbusinesses. It is important for an employer to find employees that “fit”open positions. The processes associated with finding employees that fitwell can be expensive and time consuming for an employer. Such processescan include evaluating numerous resumes and cover letters, telephoneinterviews with candidates, in-person interviews with candidates, drugtesting, skill testing, sending rejection letters, offer negotiation,training new employees, etc. A single employee candidate can be verycostly in terms of man-hours needed to evaluate and interact with thecandidate before the candidate is hired.

Computers and computing systems can be used to automate some of theseactivities. For example, many businesses now have on-line recruitingtools that facilitate job postings, resume submissions, preliminaryevaluations, etc. Additionally, some computing systems includefunctionality for allowing candidates to participate in “virtual”on-line interviews.

The job of interviewers and candidate reviewers is to determine ifcandidates are skilled and have the qualifications required for aparticular job. In the process of doing this, they ideally compare andcontrast the qualifications of candidates. Over the years there havebeen numerous documented instances in which candidates have beenselected based on qualities or characteristics other than the skills andqualifications required for a particular job. In the Unites States andother jurisdictions across the world, when candidates are chosen on thebasis of gender, race, religion, ethnicity, sexual orientation,disability, or other categories that are protected to some degree bylaw, penalties may be imposed on entities for such practices. Thepenalties may be financial and may also include requirements formonitoring of hiring practices to ensure violations are not repeated.Additionally, when candidates are selected based on non-work relatedcharacteristics, the best candidates for the position may be overlooked,such that the quality of an entity's workforce is compromised. Whileefforts have been made in the past to avoid discriminatory practices inhiring, these efforts have not been entirely satisfactory.

The subject matter claimed herein is not limited to embodiments thatsolve any disadvantages or that operate only in environments such asthose described above. Rather, this background is only provided toillustrate one exemplary technology area where some embodimentsdescribed herein may be practiced.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings in which likereferences indicate similar elements. It should be noted that differentreferences to “an” or “one” embodiment in this disclosure are notnecessarily to the same embodiment, and such references mean at leastone.

FIG. 1 is a block diagram of an exemplary network architecture in whichembodiments of a bias detection tool may operate, according to someembodiments.

FIG. 2 is a block diagram of a bias detection tool according to someembodiments.

FIG. 3A-B illustrate graphs of processed audio signals for utteranceidentification according to some embodiments.

FIG. 4 is a graph of spectral analysis of identified utterancesaccording to some embodiments.

FIGS. 5A, 5B, 5C, and 5D illustrate a series of images from a processingof characteristic extraction from video frames, according to someembodiments.

FIG. 6 is plot showing the output of an unsupervised clustering approachto identifying bias according to some embodiments.

FIG. 7 is an exemplary graphical user interface for assessing evaluatorbias according to some embodiments.

FIG. 8 is a flow diagram of a method of assessing evaluator biasaccording to some embodiments.

FIG. 9 illustrates a diagrammatic representation of a machine in theexemplary form of a computing system for model-assisted evaluation andintelligent interview feedback according to an embodiment.

Some aspects of these figures may be better understood by reference tothe following Detailed Description.

DETAILED DESCRIPTION

Methods and systems for bias detection to improve the reviewing andassessment of digital interviews and other digitally-capture evaluationprocesses are described. In the following description, numerous detailsare set forth. In one embodiment, a bias detection tool receives a setof evaluation data from a candidate evaluation system. The evaluationdata is generated by a set of evaluators based on digital interview datafrom evaluation candidates. The bias detection tool extractscharacteristics of the evaluation candidates from the digital interviewdata, classifies the evaluation candidates based on the characteristicsof the candidate extracted from the digital interview data, anddetermines whether the evaluation data indicates a bias of one or moreevaluators of the set of evaluators with respect to one or more of theextracted characteristics. The extraction of characteristics may resultin a set of unknown characteristics as the result of an unsupervisedclustering algorithm. Or the extraction and classifying may be performedby a model that is trained with a set of known information. If a bias inevaluation is determined to be present, the bias detection tool maynotify an evaluation campaign manager, such as a human resourcesdirector. This determination may be made before the results of the biasrise to the level of justifying legal action, allowing companies andother organizations to take action against conscious or unconscious biasat an earlier stage. This bias detection tool may also assist companiesthat are under court-directed orders to eliminate bias in hiringpractices.

In some instances in this description, well-known structures and devicesare shown in block diagram form, rather than in detail, in order toavoid obscuring the embodiments of the present disclosure. It will beapparent, however, to one of ordinary skill in the art having thebenefit of this disclosure, that embodiments of the present disclosuremay be practiced without these specific details.

With the ability to recruit for positions nationally and eveninternationally using the Internet, the number of qualified candidatesapplying for a given job can be expensive and time consuming toevaluate. For more technical positions, subject-matter experts are usedfor evaluation and screening of candidates rather than focusing onregular job duties. With the adoption of digital video interviewing, thetime needed to evaluate candidates is reduced, however, the problem ofhaving too many candidates to filter through still remains.

Digital interviews or other evaluations, such as a pitch for investmentfunding or a grant, an admissions interview, job performanceevaluations, or other presentation meriting assessment and comparisonmay include responding to a series of prompts or questions. Theresponses to those prompts by a person or group being evaluated can becaptured as digital data and later reviewed and rated by an evaluator.Because there are many candidates, a large data set is collected thatincludes the recorded responses for each candidate. When evaluatorslater view the recorded responses, the evaluators may provide ratingsfor each response or for some responses and may also providerecommendations as to the final evaluation decision. For example,evaluators may rate responses on a scale, such as zero to five, and mayprovide recommendations, such as “yes,” “no,” “maybe,” etc. When ratingsand/or recommendations are provided in a non-quantitative format, thoseratings and recommendations may be converted to numerical values. Forexample, the “yes” may be converted to a one, the “no” may be convertedto a zero, and the “maybe” may be converted to one-half. This mayfacilitate the application of statistical models and machine-learning inthe assessment and selection of candidates.

Because the evaluators are tasked with providing ratings for candidates'responses, there is a degree of subjectivity included in each rating.This subjectivity on the part of evaluators may, in some cases, beimpacted by the evaluators' conscious and unconscious biases. Forexample, an evaluator may be biased against candidates with an accentthat indicates a certain ethnicity. Or an evaluator may be biasedagainst a candidate, due to that candidates perceived race, religion,gender, disability, etc. This bias may be reflected in the evaluator'sratings of candidates' responses and in the evaluator's recommendations.If the magnitude of the impact is great enough, a violation of law maybe the result. However, many companies are committed to eliminating anysuch bias in their hiring practices and may want to be apprised of anybias at all, even if the bias results in disparate impact that is lessthan any limits enforced in a given jurisdiction.

By accessing volumes of digital interview data and evaluation data, thatdata may be searched and analyzed to monitor for and detect biases inthe selection process. To facilitate this, characteristics of candidatesmay be extracted by machines from the digital interview data. In someinstances, candidates may provide information that may be useful inassessing evaluator bias. For example, candidates may provideinformation regarding race, religion, gender, sexual orientation, etc.In some evaluations, such information may be requested as part of theevaluation process. Because of sensitivities and concerns surroundingthe use of such information some candidates may decline to provide thatinformation. In addition to using explicitly provided information toextract characteristics of the evaluation candidates, machine-learningmay be applied to audio and/or video information provided in the digitalinterview data to identify indicators of such characteristics. In someembodiments, candidates may be presented with a user interface elementby which the candidate may request to opt-out of any such characteristicassessment. In such circumstances, the characteristics of the candidateor candidates choosing to opt-out may not be collected. In suchcircumstances, the detection of bias may become more difficult as lessinformation is available to enable the detection to be performance.

Embodiments described herein can be used to address issues of bias inthe selection of candidates for a given position. The embodiments, mayallow users to address and eliminate biases in order to minimize theirimpact. By eliminating the impact of biases, the best candidates may beselected. In circumstances in which unlawful biases in selection havebeen identified, the embodiments described herein may allow for moreaccurate monitoring of the hiring process and may be useful indemonstrating a required change in practices in some instances.

FIG. 1 is a block diagram of a network architecture 100 in whichembodiments of a bias detection tool 110 may operate. The networkarchitecture 100 may include multiple client computing systems 102(“clients 102”) coupled to a server computing system 104 via a network106 (e.g., a public network such as the Internet, a private network suchas a local area network (LAN), or a combination thereof). The network106 may include the Internet and network connections to the Internet.Alternatively, the server 104 and the clients 102 may be located on acommon LAN, personal area network (PAN), campus area network (CAN),metropolitan area network (MAN), wide area network (WAN), wireless localarea network, cellular network, virtual local area network, or the like.The server computing system 104 (also referred to herein as server 104)may include one or more machines (e.g., one or more server computersystems, routers, gateways) that have processing and storagecapabilities to provide the functionality described herein. The servercomputing system 104 may execute a predictive model, referred to hereinas a bias detection tool 110. The bias detection tool 110 can performvarious functions as described herein and may include severalsubcomponents and features as described in more detail below withrespect to FIG. 2.

The bias detection tool 110 can be implemented as a part of a digitalevaluation platform 101, such as the digital interviewing platformdeveloped by HireVue, Inc., of South Jordan, Utah, or may be implementedin another digital evaluation platform such as an investment evaluationplatform or an admission evaluation platform. While many of the examplesprovided herein are directed to an employment/hiring context, theprinciples and features disclosed herein may be equally applied to othercontexts and so such are within the scope of this disclosure as well.For example, the principles and features provided herein may be appliedto a job performance evaluation, an evaluation of a sales pitch, anevaluation of an investment pitch, etc.

The bias detection tool 110 can be implemented as a standalonepredictive model that interfaces with the digital evaluation platform101 or other systems. It should also be noted that in this embodiment,the server computing system 104 implements the bias detection tool 110,but one or more of the clients may also include client modules of thebias detection tool 110 that can work in connection with, orindependently from the functionality of the bias detection tool 110 asdepicted on the server computing system 104.

The client computing systems 102 (also referred to herein as “clients102”) may each be a client workstation, a server, a computer, a portableelectronic device, an entertainment system configured to communicateover a network, such as a set-top box, a digital receiver, a digitaltelevision, a mobile phone, a smart phone, a tablet, or other electronicdevices. For example, portable electronic devices may include, but arenot limited to, cellular phones, portable gaming systems, wearablecomputing devices or the like. The clients 102 may have access to theInternet via a firewall, a router or other packet switching devices. Theclients 102 may connect to the server 104 through one or moreintervening devices, such as routers, gateways, or other devices. Theclients 102 are variously configured with different functionality andmay include a browser 140 and one or more applications 142. The clients102 may include a microphone and a video camera to record responses asdigital interview data. For example, the clients 102 may record andstore video responses and/or stream or upload the recorded responses tothe server 104 for capture and storage. In one embodiment, the clients102 access the digital evaluation platform 101 via the browser 140 torecord responses. The recorded responses may include audio, video,digital data, such as code or text, or combinations thereof. In suchembodiments, the digital evaluation platform 101 is a web-basedapplication or a cloud computing system that presents user interfaces tothe clients 102 via the browser 140.

Similarly, one of the applications 142 can be used to access the digitalevaluation platform 101. For example, a mobile application (referred toas “app”) can be used to access one or more user interfaces of thedigital evaluation platform 101. The digital evaluation platform 101 canbe one or more software products that facilitate the digital evaluationprocess. For example, in some cases, the one of the clients 102 is usedby a candidate (or interviewee) to conduct a digital interview. Thedigital evaluation platform 101 can capture digital response data 132from the candidate and store the data in a data store 130. The digitalresponse data 132 may include data uploaded by the candidate, audiocaptured during the interview, video captured during the interview, datasubmitted by the candidate before or after the interview, or the like.As illustrated herein, the digital response data 132 includes at leastrecorded response in the form of video captured during the interview.This digital response data 132 may be used to identify multipleindicators for use in extracting characteristics of the candidates as isdiscuss in more detail below.

The clients 102 can also be used by a reviewer or evaluator to review,screen, and select candidates and their associated response data. Thereviewer can access the digital evaluation platform 101 via the browser140 or the application 142 as described above. The user interfacespresented to the reviewer by the digital evaluation platform 101 aredifferent than the user interfaces presented to the candidates.Similarly, user interfaces presented to personnel that supervise theevaluators (herein a supervisor) may be different, as well, and maypresent more comprehensive information. The user interfaces presented tothe supervisor permit the supervisor to access the digital response data132 for reviewing and selecting the candidates based on the ratings andrecommendations of evaluators and also to receive information regardingpotential biases detected by the bias detection tool 110. The biasdetection tool 110 can be activated by the supervisor (or automaticallyactivated when so configured) to identify whether bias is likely presentin a given evaluation campaign. The bias detection tool 110 may be ableto provide information as to whether or not individual evaluatorsexhibit one or more biases and what those biases are, as indicated bythe assessment of information stored in the data store 130.

The data store 130 can represent one or more data repositories on one ormore memory devices. The data store 130 may be a database or any otherorganized collection of data. The data store 130 may store the digitalresponse data 132, evaluation ratings data 134, evaluationrecommendation data 136, indicator data 138, and campaign data 139. Theindicator data 138 may include information regarding multiple indicatorsthat may be used in estimating the characteristics of a candidate. Forexample, where the candidate has provided explicit information regardinginformation such as age, race, ethnicity, religion, gender, sexualorientation, disability, socio-economic status of the candidate orfamilial socio-economic status, citizenship status, association withinstitutions such as schools, charities, political organization, etc.,that information may be stored in the indicator data 138. Also, wheremachine-learning algorithms are used to estimate such characteristicsfrom audio and video components of the digital response data 132, as isdiscussed below in more detail, the estimated or predictedcharacteristics may be stored in the indicator data 138.

In the depicted embodiment, the server computing system 104 may executethe digital evaluation platform 101, including the bias detection tool110 for detecting potential bias in the evaluation process. The server104 can include web server functionality that facilitates communicationbetween the clients 102 and the digital evaluation platform 101 toconduct digital interviews or review digital interviews, includingrecorded responses, as described herein. Alternatively, the web serverfunctionality may be implemented on a machine other than the machinerunning the bias detection tool 110. It should also be noted that thefunctionality of the digital evaluation platform 101 for recording thedigital response data 132 can be implemented on one or more servers 104and the functionality of the digital evaluation platform 101 can beimplemented by one or more different servers 104. In other embodiments,the network architecture 100 may include other devices, such asdirectory servers, website servers, statistic servers, devices of anetwork infrastructure operator (e.g., an ISP), or the like.Alternatively, other configurations are possible as would be appreciatedby one of ordinary skill in the art having the benefit of thisdisclosure.

FIG. 2 is a block diagram of the bias detection tool 110 according tosome embodiments. The bias detection tool 110 can be implemented asprocessing logic comprising hardware (circuitry, dedicated logic, etc.),software (such as is run on a general purpose computing system or adedicated machine), firmware (embedded software), or any combinationthereof. In the depicted embodiment, the evaluation review tool 110includes a user identification module 202, a collection engine 204, agraphical user interface (GUI) engine 206, a classification module 208,and a bias detection module 210. The components of the evaluation reviewtool 110 may represent modules that can be combined together orseparated into further modules, according to some embodiments.

The user identification module 202 may be used to identify users of thedigital evaluation platform 101 and to ensure that users may only accessdata they are authorized to access. To do this, the user identificationmodule 202 may include or have access to multiple profiles for the usersthat accesses the bias detection 110. For example, access to the biasdetection tool 110 may be limited to supervisors that have a role inoverseeing evaluation campaigns. In some instances, the supervisors mayinclude court-appointed supervisors, appointed as the result of alawsuit or regulatory proceeding. A supervisor may access the digitalevaluation platform 101 and be prompted to enter credentials that, whenverified, permit the supervisor to access multiple campaigns or alimited set of campaigns. For example, the supervisor may be a hiringmanager at an information technology (IT) firm that is seeking to fillpositions in IT administration, sales, and human resources and seekingto avoid or eliminate bias in the filling of those positions. The useridentification module 202 may identify the supervisor to the digitalevaluation platform 101.

The collection engine 204 may communicate with various processors anddata stores over one or more communication channels to retrieve data foruse by the bias detection tool 110. For example, when a supervisor wantsto monitor a campaign for bias, the decision maker may select thecampaign using a user interface element. If the supervisor is concernedthat bias may have impacted multiple campaigns, the supervisor mayselect multiple campaigns. Upon selection of the campaign or campaigns,the collection engine 204 may retrieve associated evaluation data 212.For example, the collection engine 204 may communicate with the datastore 130 of FIG. 1 to retrieve ratings data 134 and recommendation data136 associated with the campaign. The collection engine 204 may alsoretrieve indicator data, such as indicator data 138 from the data store130. As shown in FIG. 2, the indicator data 138 may include categoricalindicators 209A, audio indicators 209B, and visual indicators 209C. Thecategorical indicators 209A may be obtained the information explicitlyprovided by the candidates that may indicate that the candidate may beassociated with one or more categories of groups that may be subject tobias. For example, if a candidate indicates that he is male, Jewish, andCuban-American, the categorical indicators 209A permit the biasdetection tool 110 to access and process that information. Othercategorical indicators 209A associated with a candidate may include thename of the candidate, the region or area in which the candidateresides, and/or any other candidate information that may increase theprobability of correctly classifying the candidate.

Referring now to FIGS. 3A and 3B, shown therein are graphs 300 and 310illustrating processed audio signals which may be used to generate audioindicators 209B. Audio indicators 209B include information obtained fromaudio portions in the digital response data 132 of FIG. 1. The audioportions may include utterance pitch, duration, magnitude and otherfeatures. Graph 300 shows amplitudes of identified utterances 1-12 froma raw audio file, such as may be obtained from the recorded responses ofa candidate from the digital response data 132. The large gap 302 at thebeginning is before the candidate begins to speak. The numbered sectionsrepresent speaking utterances from the candidates with a correspondingutterance identifier (1-12) at the top in the order the utterancesoccurred. For example, utterance identifiers eight and nine are examplesof filler words (e.g., uh, ah, um). Graph 310 of FIG. 3B shows a plot ofmagnitudes of the identified utterances and the corresponding utteranceidentifiers. The magnitude can be utterance lengths (e.g., in seconds orother units). Likewise, similar plots can be created for the gapsbetween the identified utterances. The digital evaluation platform 101may provide for voice-to-text conversion that identifies individualutterances and identifies corresponding words. Many differenttechnologies for converting the audio of the recorded responses to textmay be used. Additionally, the digital evaluation platform 101 mayprovide text processing to gain information from the words spoken by thecandidate. In some embodiments, the digital evaluation platform 101 mayprovide an indication as to the regional or national origin ofcandidates based on the audio file, or the processed audio data.

FIG. 4 a graph 400 of spectral analysis of identified utterancesaccording to some embodiments. The graph 400 shows the spectral analysisfor each of the identified utterances illustrated in FIG. 3B. Thespectral analysis can be used for a single word, one or more phrases, aswell as for interview fingerprinting. The y axis of graph 400 is thepower 401 and the x axis is the frequency 402. Using the same utterancesegmentation method described above, spectral analysis can be completedon each utterance. Using the utterance time series data, the processinglogic can compute summary statistics for each window within thespectrum. For example, each window may be defined by stepping 500 kHz(i.e., 1-500 kHz=window 1, 501-1000 kHz=window 2, etc.). Alternatively,other window sizes can be defined, and different frequency ranges can beevaluated. The summary statistics that were used on the spectralanalysis may include max, min, median, skew, standard deviation, mode,slop, kurtosis, or other types of summary statistics.

By using the summary statistics and other methods, the audio indicators209B may provide information that is statistically relevant inclassifying candidates according to many different categories.

Referring now to FIGS. 5A-D, shown therein is a series of imagesillustrating how some information may be obtained from video frames of acandidate's response to produce some of the visual indicators 209C. Thestill image 500 of FIG. 5A shows an exemplary image from a recordedresponse, such as may be stored in the digital response data 132. Asseen in the still image 500, the exemplary candidate 502 is shownapproximately in the middle of the field of view. Because the process ofperforming a digital interview often includes a candidate recordingresponses from home, the still image 500 also includes a dresser 504 anda smoke alarm 506. In the process of evaluation candidates, those makingdecisions are inclined to use any information made accessing during orby the interview process. In a conventional, face-to-face interview, forexample, an evaluator may be influenced by the clothes the candidatewears to the interview. Because the candidate 502 is able to recordedresponses at home, many other features, like the dresser 504 and thesmoke alarm 506 may be assessed. The environment in which candidates,like the candidate 502, are recorded may similarly influence evaluators.The environment may provide information as to candidate's socio-economicbackground. The digital evaluation platform 101 may collect informationfrom the candidate 502 and the background in the still image 500 andprocess the information to generate visual indicators 209C. Theseindicators 209A may allow for the classification of candidates based onrace, gender, ethnicity, sexual orientation, age, and socio-economicstatus.

To identify information that may be included in the visual indicators209C, the still image 500 may undergo several processing steps. Forexample, when the video is in color, the still image 500 may beconverted to a gray-scale image to facilitate certain kinds ofprocessing. As shown in FIG. 5B, the still image 500 is enhanced tocompensate for poor lighting conditions. More detail is apparent in thestill image 500 of FIG. 5B than in the still image 500 of FIG. 5A. Asshown in FIG. 5B an adaptive histogram equalization process has beenperformed to the still image 500. Many other enhancements may be used toprepare the still image 500 for more detailed image, such as facialrecognition and mapping.

After enhancement as seen in FIG. 5B, a facial region 508 is identifiedby the digital evaluation platform 101 as seen in FIG. 5C. Manydifferent techniques may be used to identify the facial region 508. Forexample, haar cascades may be used to determine the extent of the facialregion 508. After identifying the facial region 508, more detailedprocessing of the face of the candidate 502 may be performed. Again,many different facial recognition techniques may be employed. As shownin FIG. 5C, an Eigenface approach is used. This approach provides a setof eigenvectors which may be compared with the still image 500 to detectthe face. When the face is detected, the facial region 508 or a portionthereof may be provided to a secondary facial recognition systemprovided by the digital evaluation platform 101 for a similarity lookup.

With the face isolated, an Active Appearance Model (AAM) may be used tomatch and analyze the facial features of the candidate 502. The ActiveAppearance Model is a computer vision algorithm that includes a numberof points that may be mapped onto the face to form a model of the face.Referring now to FIG. 5D, shown therein a close-up view of the facialregion 508As shown in FIG. 5D, the model 510 corresponds to the face ofthe candidate 502 and provides relative spacing between identifiedfeatures. The relative spacings of the model 510 may be used to providesome of the visual indicators 209C. Additionally, the model 510 may beused to identify areas of the head and face of the candidate 502 thatmay be analyzed in various ways. For example, an eye color of thecandidate 502 may be obtained. Additional visual indicators 209C mayinclude indicators for skin tone and hair color in addition to eyecolor. These tones and colors may be expressed as individualred-green-blue (RGB) component values.

The visual indicators 209C may be combined with the categoricalindicators 209A and the audio indicators 209B to produce a combinedvector representation of multiple candidates, with each row representinga single candidate “n” in a matrix X_(n). An example of such a matrixX_(n) is seen below:

As seen in the matrix X_(n), each row includes indicators 209A-C for asingle candidate. Only some of the indicators 209A-C are shown in matrixX_(n). As shown in the matrix X_(n) above, the components are expressedas scaled values from zero to one. The normalization and scaling ofinputs to the matrix X_(n) may reduce the likelihood that a large inputmay have on the model relative to a smaller input value. Each column maybe scaled by its maximum value in some embodiments. The components ofthe matrix X_(n) may be understood as characteristics of the evaluationcandidates. Thus, the median green skin value of candidate may be one ofmany characteristics extracted from the indicators 209A-C for a givencandidate.

Another matrix, matrix Y, may be provided for training purposes. In someembodiments, the matrix Y may be obtained from some of the categoricalindicators 209A provided directly by the candidates. In someembodiments, the matrix Y includes multiple columns, one for eachclassification of candidates. Thus, embodiments of the matrix Y mayinclude a column for gender or sex, race, age, etc. In some embodiments,the matrix Y may include only a single column with an entry for each ofthe candidates in the matrix X_(n), as shown below:

In some embodiments, the age values may be estimated by evaluators, andthus may be indicated as part of a range of ranges rather than specificages.

Given the matrices X_(n) and Y, a model may be identified and trainedusing various methods such as regression, support vector machines, deeplearning, genetic programming, or another suitable regression ofclassification technique.

These matrices X_(n) and Y may be provided to the classification module208, which receives characteristics of the matrix X_(n) as an input toclassify the candidates whose characteristics are included in the matrixX_(n). When the matrix X_(n) and Y are provided are a historical dataset for training purposes, the classification module 208 may provide themodel by which subsequent characteristics from candidates may be used toclassify those candidates. Because the classifications may be providedexplicitly in such circumstances, this may be understood as a supervisedlearning approach.

In some embodiments, an unsupervised learning approach may be used. Insuch an approach, the Y matrix may not be included or may be included aspart of the input data. In such an approach, the classification module208 may receive in the input set and cluster similar individuals usingk-means clustering, k-harmonic means, k-harmonic means withoptimization, or another unsupervised classification algorithm. Theclassification module 208 may also receive a cluster number input, whichindicates the number of clusters that are permitted by theclassification module 208 in performing the classification ofcandidates. An example of such an unsupervised learning approach may befound in the plot 600 of FIG. 6.

As shown in FIG. 6, the plot 600 includes two clusters, cluster 1 andcluster 2. Thus, the classification module 208 received input indicatingthat two clusters should be produced from the modelling process. Themodelling process produces the most decisive clustering into twoclusters. The unsupervised learning approach may allow for theidentification of untracked physical biases, such as personal weight orhair color. After the clustering by the classification module 208 isfinalized, a supervisor may view members of each cluster to assess thequality or qualities upon which the clusters 1 and 2 are based. Forexample, as seen in plot 600, the candidates are clustered according towhether they are male or female. When more than two clusters arerequested as input to the classification module 208, qualities that arenot binary may be the basis for clustering.

After the classification module 208 receives the indicators 209A, 209B,and 209C, the classifications may be provided to the bias detectionmodule 210 of the bias detection tool 110. The bias detection tool 110may process the indicators 209A-C and the evaluation data 212 todetermine whether a bias is present on the part of at least one of theevaluators with respect to at least one of the characteristics of thecandidates. When statistical support is found to suggest that a bias ispresent, the bias detection module 210 may provide a notification to asupervisor that is associated with the particular evaluation campaign inthe campaign data 139 of FIG. 1. The notification may be setup by thesupervisor or by default to be sent in the form of an email or atext-message to the supervisor or to other recipients. Additionally, theGUI engine 206 may display a notification in a user interface providedfor the bias detection tool 110 and the digital evaluation platform 101.Thus, when an evaluator is determined to be outside the mean by athreshold value, a notification may be generated and sent. The thresholdvalue may be a dynamic value that adjusts based on a variety of factors,such as sample size, the available candidate pool, and/or a confidencevalue.

FIG. 7 illustrates an exemplary graphical user interface 700 forassessing evaluator bias according to some embodiments. The userinterface 700 includes a bias assessment window 702, and a drop-downselector element 704 that permits a supervisor, a compliance offer, oranother person with access to the bias detection tool 110 to select anindividual position, a position category (as shown in FIG. 7), adivision, or an entire company for bias detection analysis. Asillustrated in FIG. 7, the bias assessment window 702 displays relevantdata for the position category of “developer.” The bias detection tool110 can compare the evaluation behaviors of a set of evaluatorsaccording to various criteria and assess how similar evaluators'recommendations and/or ratings are.

The user interface 700 includes a plot 710 that compares multipleevaluators in terms of their exhibited bias and their similarity toother evaluators. As shown in FIG. 7, the plot 710 is based on therecommendations provided by the evaluators, which is accessed by thebias detection module 210 of the bias detection tool 110 from theevaluation data 212. The plot 710 may be a two-dimensional plot, or mayfurther include a third-dimension of data by varying the points used torepresent evaluators on the graph. The x-axis 712A of the plot 710orders the points, such as exemplary points 714A, 714B, and 714C, thatrepresent evaluators according to how biased the evaluator is. They-axis 712B orders the points according to the similarity of theevaluators.

Each of the evaluators listed in the evaluator list 720, as illustratedin FIG. 7 is represented by a point in the plot 710. The evaluatorrepresented by point 714A exhibits slightly less bias than the evaluatorrepresented by point 714A, but is much less similar to the otherevaluators. The plot 710 also includes a threshold line 716. Becausemost evaluators will exhibit a degree of bias in some aspects, thethreshold line 716 may be used to indicate the amount of bias that maynot require action, for example, an evaluator may be biased in favor offellow alums. As shown in FIG. 7, the evaluator represented by point714C is to the right of the threshold line 716. A supervisor orcompliance offer may be prompted by a user interface element provided bythe GUI engine 206 to comment. The supervisor may comment withinformation explaining actions to be taken or actions that are to betaken to address the bias. Or the supervisor may comment as to why thedetected bias does not require action to be taken.

When a user selects either a name on the evaluator list 720 or a pointon the plot 710, a tooltip may be provided by the GUI engine 206 thatprovide information from the evaluation 212 of other data accessible onthe data store 130.

FIG. 8 is a flow diagram illustrating a method 800 for detecting bias ina set of evaluation data, according to some embodiments of the presentdisclosure. The method 800 may be performed by processing logic thatcomprises hardware (e.g., circuitry, dedicated logic, programmablelogic, microcode, etc.), software (e.g., instructions run on aprocessing device to perform hardware simulation), or a combinationthereof.

For simplicity of explanation, the method 800 and other methods of thisdisclosure may be depicted and described as a series of acts oroperations. However, operations in accordance with this disclosure canoccur in various orders and/or concurrently, and with other acts notpresented and described herein. Furthermore, not all illustrated actsmay be required to implement the methods in accordance with thedisclosed subject matter. In addition, those skilled in the art willunderstand and appreciate that the methods could alternatively berepresented as a series of interrelated states via a state diagram orevents. Additionally, it should be appreciated that the methodsdisclosed in this specification are capable of being stored on anon-transitory, tangible, computer-readable medium to facilitatetransporting and transferring such methods to computing devices.

Thus, FIG. 8 illustrates an embodiment of the method 800, which beginsat block 802 in which the processing receives evaluation data from acandidate evaluation system. The evaluation data may be provided by aset of evaluators based on digital interview collected from evaluationcandidates. The evaluation data may be the evaluation data 212 of FIG.2, which may include the ratings data 134 and the recommendation data136 as illustrated in FIG. 1. In some embodiments, the evaluation datamay include the digital response data 132. This information may bereceived by the bias detection module 210 of the bias detection tool110.

At block 804, the processing logic may extract characteristics of theevaluation candidates from the digital interview data. For example,characteristics may be extracted from the digital interview data, suchas the digital interview data response data 132. As described herein,categorical indicators 209A may include indicators provided explicitlyby the evaluation candidates. Such indicators may include names, placesof residence, identifiers of devices use in providing recorded responsesand/or written responses to the digital evaluation platform 101, etc.Other indicators include audio indicators 209B, which may include pitch,speech rate, and accent, in addition to text obtained from the audiodata of the recorded responses using voice-to-text technologies. Visualindicators 209C may also be included. The visual indicators 209C mayinclude relative dimensions or spacings of facial features of theevaluation candidates, as well as information regarding the skin tone,hair color, and eye color of evaluation candidates. The indicators209A-C may be provided to the classification module 208 of the biasdetection tool 110, which may process the indicators.

At block 806, the processing logic classifies the evaluation candidatesbased on the characteristics of the candidate extracted from the digitalinterview data. This may be done by the classification module 208 inmultiple ways as described herein. For example, the indicators 209A maybe provided to a trained model provided by the classification model. Themodel may have been trained earlier using a set of indicatorsrepresented by matrix and a set of classifications Y, as describedherein. After the model of the classification module 208 is trained,indicators associated with specific candidates may be used to classifythe candidates according to race, gender, ethnicity, sexual orientation,age, socioeconomic status, etc. In some embodiments, unsupervisedlearning algorithms may be used. As shown in plot 600 of FIG. 6, the setof indicators may be provided to a clustering algorithm along with aninput to define the number of clusters. The unsupervised learningalgorithms provided by the classification module 208 may sort thecandidates into clusters, which may then be assessed by a supervisor todetermine the characteristics upon which the clustering occurred. Forexample, the clustering may occur based on whether candidates are maleor female as shown in plot 600. In some embodiments, candidates may beclustered according to race or another non-binary value. Using either asupervised (training-based) algorithm or an unsupervised algorithm, theclassification module 208 may receive the indicators for the set ofcandidates and classify the candidates in terms of classifications thatmay be subject to bias in the evaluation process.

At block 808, the processing logic may determine whether the evaluationdata indicates a bias of one or more evaluators of the set of evaluatorswith respect to one or more of the classifications of the evaluationcandidates. This may be done based on statistical modeling of thevarious classifications of the candidates. For example, the four-fifthsrule may be used by determining the classification in a category, suchas race, that receives the highest recommendations and checking to seewhether other race classifications perform at least 80% as well in therecommendations as that group. Many similar tests may be used. Forexample, if an age classification, such as candidates estimated to beover 50 years old, receives statistically significantly lower marks onratings of by a particular evaluator, this evaluator may be flagged to asupervisor as having a potential bias. By having information regardingpotential bias, interventions may be undertaken to address and eliminateany biases. As described herein, when a potential bias is detected bythe bias detection tool 110, the bias detection tool 110 may provide anotification to a supervisor or another party regarding the potentialbias. In some embodiments, the GUI engine 206 of the bias detection tool110 may provide for a comment user element by which a supervisor orcompliance officer may comment on the potential bias. The comments mayindicate actions taken or may indicate why the potential bias is notactual bias.

The bias detection tool 110, and its components as described herein, canbe used to assess potential biases that may be introduced by evaluatorsin the subjective aspects of a digital evaluation process. The biasdetection tool 110 may prevent a company that has faced problems of biasfrom repeating such problems and may allow the company, or athird-party, to easy monitor the company's performance in this regard.Companies that have not faced problems, perceived or actual, withdifferent types of bias may be able to detect potential bias and addressit early.

FIG. 9 illustrates a diagrammatic representation of a machine in theexemplary form of a computing system for bias detection. Within thecomputing system 900 is a set of instructions for causing the machine toperform any one or more of the methodologies discussed herein. Inalternative embodiments, the machine may be connected (e.g., networked)to other machines in a LAN, an intranet, an extranet, or the Internet.The machine may operate in the capacity of a server or a client machinein a client-server network environment, or as a peer machine in apeer-to-peer (or distributed) network environment. The machine may be aPC, a tablet PC, a set-top-box (STB), a personal data assistant (PDA), acellular telephone, a web appliance, a server, a network router, switchor bridge, or any machine capable of executing a set of instructions(sequential or otherwise) that specify actions to be taken by thatmachine. Further, while only a single machine is illustrated, the term“machine” shall also be taken to include any collection of machines thatindividually or jointly execute a set (or multiple sets) of instructionsto perform any one or more of the methodologies discussed herein forbias detection, including classification of candidates, for evaluatingdigital interviews and other assessment or evaluations and theevaluators for bias, such as embodiments of the method 800 as describedabove. In one embodiment, the computing system 900 represents variouscomponents that may be implemented in the server computing system 104 asdescribed above. Alternatively, the server computing system 104 mayinclude more or less components as illustrated in the computing system900.

The exemplary computing system 900 includes a processing device 902, amain memory 904 (e.g., read-only memory (ROM), flash memory, dynamicrandom access memory (DRAM) such as synchronous DRAM (SDRAM), etc.), astatic memory 906 (e.g., flash memory, static random access memory(SRAM), etc.), and a data storage device 916, each of which communicatewith each other via a bus 930.

Processing device 902 represents one or more general-purpose processingdevices such as a microprocessor, central processing unit, or the like.More particularly, the processing device 902 may be a complexinstruction set computing (CISC) microprocessor, reduced instruction setcomputing (RISC) microprocessor, very long instruction word (VLIW)microprocessor, or a processor implementing other instruction sets orprocessors implementing a combination of instruction sets. Theprocessing device 902 may also be one or more special-purpose processingdevices such as an application specific integrated circuit (ASIC), afield programmable gate array (FPGA), a digital signal processor (DSP),network processor, or the like. The processing device 902 is configuredto execute the processing logic (e.g., bias detection tool 926) forperforming the operations and steps discussed herein.

The computing system 900 may further include a network interface device922. The computing system 900 also may include a video display unit 910(e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), analphanumeric input device 912 (e.g., a keyboard), a cursor controldevice 914 (e.g., a mouse), and a signal generation device 920 (e.g., aspeaker).

The data storage device 916 may include a computer-readable storagemedium 924 on which is stored one or more sets of instructions (e.g.,bias detection tool 926) embodying any one or more of the methodologiesor functions described herein. The bias detection tool 926 may alsoreside, completely or at least partially, within the main memory 904and/or within the processing device 902 during execution thereof by thecomputing system 900, the main memory 904 and the processing device 902also constituting computer-readable storage media. The bias detectiontool 926 may further be transmitted or received over a network via thenetwork interface device 922.

While the computer-readable storage medium 924 is shown in an exemplaryembodiment to be a single medium, the term “computer-readable storagemedium” should be taken to include a single medium or multiple media(e.g., a centralized or distributed database, and/or associated cachesand servers) that store the one or more sets of instructions. The term“computer-readable storage medium” shall also be taken to include anymedium that is capable of storing a set of instructions for execution bythe machine and that causes the machine to perform any one or more ofthe methodologies of the present embodiments. The term“computer-readable storage medium” shall accordingly be taken toinclude, but not be limited to, solid-state memories, optical media,magnetic media or other types of mediums for storing the instructions.The term “computer-readable transmission medium” shall be taken toinclude any medium that is capable of transmitting a set of instructionsfor execution by the machine to cause the machine to perform any one ormore of the methodologies of the present embodiments.

The bias detection tool, components, and other features described hereincan be implemented as discrete hardware components or integrated in thefunctionality of hardware components such as ASICS, FPGAs, DSPs, orsimilar devices. The bias detection module 932 may implement operationsof bias detection as described herein. In addition, the bias detectionmodule 932 can be implemented as firmware or functional circuitry withinhardware devices. Further, the bias detection module 932 can beimplemented in any combination hardware devices and software components.

Some portions of the detailed description that follow are presented interms of algorithms and symbolic representations of operations on databits within a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of steps leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, characters,terms, numbers or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the following discussion,it is appreciated that throughout the description, discussions utilizingterms such as “receiving,” “generating,” “analyzing,” “capturing,”“executing,” “extracting,” “specifying,” “selecting,” “classifying,”“processing,” “providing,” “computing,” “calculating,” “determining,”“displaying,” or the like, refer to the actions and processes of acomputing system, or similar electronic computing systems, thatmanipulates and transforms data represented as physical (e.g.,electronic) quantities within the computing system's registers andmemories into other data similarly represented as physical quantitieswithin the computing system memories or registers or other suchinformation storage, transmission or display devices.

Embodiments of the present disclosure also relate to an apparatus forperforming the operations herein. This apparatus may be speciallyconstructed for the required purposes, or it may comprise ageneral-purpose computing system specifically programmed by a computerprogram stored in the computing system. Such a computer program may bestored in a computer-readable storage medium, such as, but not limitedto, any type of disk including optical disks, CD-ROMs, andmagnetic-optical disks, read-only memories (ROMs), random accessmemories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any typeof media suitable for storing electronic instructions.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the scope of the disclosure to the precise forms disclosed. Manymodifications and variations are possible in view of the aboveteachings. The embodiments were chosen and described in order to bestexplain the principles of the disclosure and its practical applications,to thereby enable others skilled in the art to utilize the disclosureand various embodiments with various modifications as may be suited tothe particular use contemplated.

1. A method comprising: receiving, by a human bias detection toolexecuting by a processing device, evaluation data from a candidateevaluation system, the evaluation data provided by a set of evaluatorsbased on digital interview data collected from evaluation candidates,wherein the digital interview data comprises video frames of theevaluation candidates; performing, by the human bias detection tool,video analysis on the video frames to identify faces of the evaluationcandidates; performing, by the human bias detection tool, a computervision algorithm to build models for the evaluation candidates, thecomputer vision algorithm mapping a number of points onto the respectiveface of the respective evaluation candidate to form the respective modelfor the respective evaluation candidate; extracting, by the human biasdetection tool, indicators of human characteristics of the evaluationcandidates from the digital interview data, wherein the indicatorscomprise visual indicators, and wherein the extracting the indicatorscomprises identifying a respective visual indicator from the respectivemodel for the respective evaluation candidate; classifying, by theprocessing device, the evaluation candidates based on the indicatorsextracted from the digital interview data; and determining, by theprocessing device, whether the evaluation data indicates a bias of oneor more evaluators of the set of evaluators with respect to aclassification of the evaluation candidates, wherein the determiningwhether the evaluation data indicates a bias comprises performingstatistical modeling of the indicators of human characteristics.
 2. Themethod of claim 1, further comprising providing information regardingthe bias to an evaluation supervisor.
 3. The method of claim 2, whereinthe information regarding the bias indicates a potential violation of ananti-discrimination law.
 4. The method of claim 2, wherein theinformation regarding the bias indicates a deviation from a modeledoutcome, the modeled outcome modelling results of a set of evaluatorsnot having the bias.
 5. The method of claim 1, wherein classifying theevaluation candidates based on the indicators comprises: receiving aninput specifying a quantity of clusters permitted in a model; andapplying an unsupervised classification algorithm to the evaluation dataand the indicators to sort the evaluation candidates into the quantityof clusters.
 6. The method of claim 1, wherein the bias is with respectto at least one of gender, race, ethnicity, age, disability, orsocioeconomic status.
 7. The method of claim 1, wherein the extractingthe indicators of human characteristics of the evaluation candidatescomprises merging the visual indicators with at least one of categoricalindicators or audio indicators.
 8. The method of claim 1, whereinclassifying the evaluation candidates based on the indicators extractedfrom the digital interview data comprises providing the indicators to anevaluation classification model.
 9. A computing system comprising: adata storage device to store a set of evaluation data generated by a setof evaluators based on digital interview data, wherein the digitalinterview data comprises video frames of evaluation candidates; and aprocessing device, coupled to the data storage device, to execute anbias detection tool to: perform video analysis on the video frames toidentify faces of the evaluation candidates; perform a computer visionalgorithm to build models for the evaluation candidates, the computervision algorithm mapping a number of points onto the respective face ofthe respective evaluation candidate to form the respective model for therespective evaluation candidate; extract indicators of humancharacteristics of the evaluation candidates from the digital interviewdata, the indicators comprising visual indicators extracted from themodels for the evaluation candidates; classify the evaluation candidatesbased on the indicators extracted from the digital interview data; anddetermine whether the evaluation data indicates a bias of one or moreevaluators of the set of evaluators with respect to a classification ofthe evaluation candidates, wherein the bias detection tool is to performstatistical modeling of the indicators of human characteristics withrespect to the classification to detect whether the evaluation dataindicates the bias.
 10. The computing system of claim 9, wherein thebias detection tool is further to communicate information regarding thebias to an evaluation supervisor, the information regarding the biasindicating a deviation from a modeled outcome, the modeled outcomemodeling results of a set of evaluators not having the bias.
 11. Thecomputing system of claim 9, wherein the bias detection tool is furtherto: receive an input specifying a quantity of clusters permitted in amodel; and apply an unsupervised classification algorithm to theevaluation data and extracted indicators to sort the evaluationcandidates into the quantity of clusters.
 12. The computing system ofclaim 9, wherein the bias is with respect to at least one of gender,race, ethnicity, age, disability, or socioeconomic status.
 13. Thecomputing system of claim 9, wherein the bias detection tool is furtherto merge the visual indicators with at least one of categoricalindicators or audio indicators.
 14. A non-transitory computer-readablestorage medium storing instructions that, when executed by a processingdevice, cause the processing device to perform operations comprising:receiving, by the processing device, evaluation data from a candidateevaluation system, the evaluation data provided by a set of evaluatorsbased on digital interview data collected from evaluation candidates,wherein the digital interview data comprises video frames of theevaluation candidates; performing, by the processing device, videoanalysis on the video frames to identify faces of the evaluationcandidates; performing, by the processing device, a computer visionalgorithm to build models for the evaluation candidates, the computervision algorithm mapping a number of points onto the respective face ofthe respective evaluation candidate to form the respective model for therespective evaluation candidate; extracting, by the processing device,indicators of human characteristics of the evaluation candidates fromthe digital interview data, wherein the indicators comprise visualindicators, and wherein the extracting the indicators comprisesidentifying a respective visual indicator from the respective model forthe respective evaluation candidate; classifying, by the processingdevice, the evaluation candidates based on the indicators extracted fromthe digital interview data; and determining, by the processing device,whether the evaluation data indicates a bias of one or more evaluatorsof the set of evaluators with respect to a classification of theevaluation candidates, wherein the determining whether the evaluationdata indicates a bias comprises performing statistical modeling of theindicators of human characteristics with respect to the classification.15. The non-transitory computer-readable storage medium of claim 14,wherein the operations further comprises providing information regardingthe bias to an evaluation supervisor.
 16. The non-transitorycomputer-readable storage medium of claim 15, wherein the informationregarding the bias indicates a deviation from a modeled outcome, themodeled outcome modelling results of a set of evaluators not having thebias.
 17. The non-transitory computer-readable storage medium of claim14, wherein classifying the evaluation candidates based on theindicators comprises: receiving an input specifying a quantity ofclusters permitted in a model; and applying an unsupervisedclassification algorithm to the evaluation data and extracted indicatorsto sort the evaluation candidates into the quantity of clusters.
 18. Thenon-transitory computer-readable storage medium of claim 14, wherein thebias is with respect to at least one of gender, race, ethnicity, age,disability, or socioeconomic status.
 19. The non-transitorycomputer-readable storage medium of claim 14, wherein the extracting theindicators of human characteristics of the evaluation candidatescomprises merging the visual indicators with at least one of categoricalindicators or audio indicators.
 20. The non-transitory computer-readablestorage medium of claim 14, wherein classifying the evaluationcandidates based on the indicators extracted from the digital interviewdata comprises providing the indicators to an evaluation classificationmodel.